Problems in test environments

In automation projects, it often happens that many different environments are being used. An organisation may have one or more development environments, one or more test environments, a production environment with a fallback environment and sometimes also several maintenance environments. In this situation, the following problems might emerge:

• Returning defects. A defect detected in version X is solved in version X+1 but suddenly reoccurs in version X+2.
• No guarantee that it still works. The development team cannot guarantee that everything still works despite the fact that the release covers only a limited number of defects.
• Unannounced new features. When testing a new version, it is found that specific features (new functionality, specific technical aspects) have already been realised while the testers are not aware of them.
• No connection between defect and environments. A defect detected in environment X does not occur in environment Y while they seem to be the same environments. E.g. a defect does present itself in the acceptance test environment, but not in the system test environment.
• Defects cannot be investigated. A defect can not be investigated anymore because a user other than the tester has modifi ed the test environment.

There are two solutions to prevent these problems. In the first place, the environments must be separated according to the DTAP model. That model and how it can be used is explained in the next section "DTAP model". In the second place, formal processes must manage the setting up and maintenance of the environments. Section “Processes in test environments” discusses these management processes and how they must be organised.